Network diagnostic and monitoring tools are a common feature of modern communication systems. These tools monitor all aspects of networks, from the number of people accessing the network resources to possible attacks on the network. Network attacks are the most serious forms of network issue; therefore, more resources are dedicated to identifying them.
A common method of detecting attacks comes from intrusion detection systems (“IDS”). An IDS may detect an event based on a pre-defined criteria, but this requires the data containing the attack to pass through the IDS. Further, because the IDS must have a pre-defined criteria, identifying new variants of an attack are less likely to be detected by an IDS.
Another device used in networks is the data harvester. Data harvesters collect information about the network as the network operates. This information is typically used for reporting and may be used for forensic analysis of network issues after the fact. Unlike an IDS, which needs information specifically routed as part of the chain in the network, the data harvester typically looks at a larger portion of the entire network, since it is deployed to support normal network monitoring and planning.
The differences between an IDS and a data harvester may be shown by example. An IDS might be incapable of early warnings from an attack that does not pass though an IDS for the initial attack. The attack could be based on anything, from computer programs to external computers logging in. The IDS will eventually detect the attack, but an indeterminate amount of time could lapse until the attack comes into the IDS' limited field of view. Once the attack has been detected, the earlier activity of the attack may be forensically examined from the data harvester, but the data harvester is merely a post-attack resource and does not actually announce or identify an anomaly or attack-the data harvester only logs the activity to be later analyzed.
While the data harvester may typically see more of the network traffic than an IDS, there has not been a way to use effectively the information gathered by the data harvester in a prompt manner to alert someone of a potential attack or anomaly. This leaves a wealth of information unused for anomaly detection.
To use this currently untapped information in the data harvester, there is a need for a process that may evaluate the logged data as the data is logged and identify potential anomalies or attacks. Typical detection tools do not use preceding events to set dynamically threshold alert values. If the threshold level is set too low, more false alarms are triggered, consequently making each subsequent alarm seem less important. Conversely, if the thresholds are set too high, many potential anomalies or attacks will go unreported or will be reported too late for an effective response to be implemented.
Another threshold issue comes from network use over the course of a fixed time. Network usage typically is highest during the workday and lowest at non-work hours (nights and weekends). The same threshold level for all time results in over-reports during the workday and underreports during non-work hours. Even if these thresholds change as a function of the time of the day or week, the earlier issues of fixed thresholds still exist.
Another problem with IDS detection comes from their detailed analysis of data passing through them. An IDS typically looks at the full contents of the data, examining the data bit by bit. This allows for very detailed analysis, but is resource intensive. When attacks occur, they generally have attributes that may be detected without the need for such a detailed examination. Such attacks may be detected by taking a wider or broader view of the information rather than looking into the contents of each data packet. While the wider or broader view forms of detection may not detect the most sophisticated of attacks, such view enables rapidly detecting more common attacks before they impede the network.
Accordingly, there exists a need to use effectively the information collected by data harvesters. In order for this to occur, there needs to be a way to determine what is typical for a particular network and a way to determine if recently observed patterns are atypical.